The relationship between the organization of synaptic connections and how the brain processes information has traditionally been poorly understood. For some reflex responses the connectivity has been discovered by direct observation, and some theoretical networks have been proposed to explain other simple neural responses. Deriving a neural network's behavior requires some assumptions about the behavior of the network's components. Many models have been proposed for neuron responses, most of which fall into one of two categories. One is the pulse model, such as the model of McCulloch and Pitts (A logical calculus of the ideas immanent in nervous activity. Bulletin of Mathematical Biophysics 5: 115-133 (1943)) and the “integrate-and-fire” model. The second is the firing rate model, such as that proposed by Hopfield (Neurons with graded response have collective computational properties like those of two-state neurons. Proceedings of the National Academy of Sciences 81: 3088-3092. (1984)). However, a general problem with traditional models is that it is uncertain whether the assumptions hold for real neurons. The more detailed the assumptions are, the greater the uncertainty.
To take just one example, a standard model for neuron response assumes that activation is a nonlinear function of a weighted sum of the inputs. This function may appear to be fairly general, but it cannot express quite simple functions of two or more variables or even produce reasonable approximations of them. For example, a possible neuron response to excitatory inputs X and Y is R[S(X)+S(Y)], where S is a sigmoid function that amplifies large inputs and reduces small ones, and R restricts outputs asymptotically below some physical bound on a neuron's maximum response. Because of the nonlinearity of S, S(X)+S(Y) cannot be expressed as a weighted sum of X and Y. This implies the response function R[S(X)+S(Y)] cannot be expressed as a nonlinear function of a weighted sum of X and Y.